Convolutional Color Constancy Supplemental Material

Though we use conventional (batch) L-BFGS to finalize optimization when training our model, optimization can be sped up by using stochastic (non-batch) gradient descent techniques prior to L-BFGS. Inspired by recent advances in “second-order” stochastic gradient descent techniques [2, 6, 7], we developed a novel variant of stochastic gradient descent based on exponential decaying different quantities at different rates, which we found to work well on our task. Pseudocode for our “exponential decay SGD” technique can be seen in Algorithm 1. This technique is similar to RMSProp [6] and AdaDelta [7], in that we maintain a moving average model of the gradient-squared, and then divide the gradient by the square-root of the average-gradientsquared before taking a gradient-descent step. But in addition to maintaining a moving average gradient-squared using exponential decay, we also maintain a moving average estimate of the gradient and of the loss. The moving average gradient serves as an alternative to commonly used “minibatches”, where instead of computing the average gradient of n datapoints before taking a gradient descent step, we compute n different gradients and take n different gradient descent steps while averaging in past gradient estimates to prevent dramatic jumps during optimization. This appears to help optimization, especially in our domain where our training set sizes are somewhat small. Many SGD techniques use an adaptive learning rate, where the learning rate is increased every epoch if optimization succeeds (ie, the loss decreases) and the learning rate is decreased if optimization fails (ie, the loss is increasing or oscillating). To generalize this idea we maintain a moving average of the loss for the entire dataset and compare every sampled datapoint’s loss to that average. If a datapoint’s loss appears is less than the average we slightly increase the step size, otherwise we slightly decrease the step size. In contrast to a per-epoch learning rate revision, this approach allows the step size to vary quickly during just a single epoch of optimization, thereby speeding up optimization. We parametrized our technique in terms of half-life rather than using decay multipliers, which makes these parameters easier to reason about. For example, we found if effective to set the half-life for the average loss to be roughly the size of the dataset, so that the average loss is always a reflection of the entire dataset. The half-life of the gradient we set to be small — about the size of a mini-batch, and the half-life of the gradient-squared we set to be significantly larger than that of the gradient but less than that of the loss. To ensure that our running average estimates are correct even at the beginning of optimization, we model each moving average as the ratio of two quantities. Though we describe our algorithm as randomly sampling datapoints until convergence, in practice we optimize for a fixed number of epochs (in our experiments, 50) and for each epoch we randomly order our data and then sample every datapoint in that random order, thereby improving the coverage of our training data.